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BACKGROUND 


In February 1991, the Evaluation Sub-Committee of the Research Advisory Committee discussed the 
proposed development of an Evaluation Consultation Programme as a new community-based SPRC initiative. 
Much of the initial focus was on issues of planning and implementation of this Trillium Foundation funded 
project. In its deliberations, the Sub-Committee also recognized a need for the Council to have a more 


formalized evaluation process of its own. 


Results from a Fall 1990 Board/Volunteer/Staff retreat emphasized the need to acknowledge the link 
between effective and comprehensive evaluation and maximized planning and priority-setting capabilities. 
Subsequent Board discussion in the Spring of 1991 addressed this concern more specifically by asking: 

1. How important is it for the SPRC to continuously assess its impact(s) on the 

community and how should it be done? 

Ze How can the SPRC better integrate community input into its own process of 

evaluation? 

3. How rigid has our process been and what principles might guide us towards 


effective evaluation? 


In September 1991, the Board created an ad hoc committee, herein known as the Evaluation Task Force, 
to examine these and other questions and to prepare a report for the presentation of key findings and 


recommendations. 


EVALUATION TASK FORCE - MANDATE AND STRUCTURE 


As a first step, the Task Force developed its mandate including terms of reference and membership 


considerations. 


1. Terms of Reference 


The Evaluation Task Force will function under the following terms: 


jh To update an inventory of SPRC activities in the past two years. 

Zi To establish the relationship between programme activities and the Mission, and stated goals of the 
SPRC. 

3: To identify the relative importance of external organizations (stakeholders) in determining SPRC 
activities. 

4, To develop measurable objectives for SPRC goal statements which can serve as a basis for the ongoing 


evaluation of SPRC programs. 


5: To review the range of current and potential approaches to program evaluation within the SPRC. 


2. Membership 


Task Force committee membership was to comprise: three members from the current Board of 
Directors, two Research Advisory Committee members, two Community Development Discussion Group 


members, three Consumers/Community group representatives, and one staff person (refer to Appendix A). 


TASK FORCE ACTIVITIES 


Four primary tasks were undertaken by the committee: 
1. Analysis of an inventory of project/consultations; 

2. Monthly Staff Time Analysis - Discussion; 

3. SPRC Evaluation Matrix; and 
4 


. Developing principles for evaluation. 


1. Inventory of Projects and Consultations 
An overview of projects and major consultations completed or underway over the past 12 months was 


constructed. This also included details regarding the point-of-origin for each activity, i.e., where the initial 


proposal or recommendation for the project/consultation originated (refer to Appendix B). 


2. Monthly Staff Time Analysis 


This involved discussion regarding the continuing practice of monthly staff time analysis in the form of 
categorized time summaries for professional staff. Each summary represents the number of hours a staff person 
has contributed to various in-house committees or projects, external community consultations, or other 


involvements (refer to Appendix C). 


3. SPRC Evaluation Matrix 

This was an educational exercise intended to provide Task Force members with an example of the way 
in which SPRC activities can be categorized according to key program goal areas and corresponding evaluation 
design to measure related impacts. The four goal areas included: policy generation, community-development- 


based planning, research, and trends analysis (refer to Appendix D). 


4. Developing Principles for Evaluation 


Concurrent with an analysis of SPRC programme and evaluation resources was the development of a 
draft set of basic principles to guide SPRC evaluation activities. These principles were to apply both within the 


organization and in its many community-based activities. 


KEY FINDINGS 


1. Principles for Evaluation 


The purpose of evaluation is to gather information about the organization which will be useful in 
measuring goals and objectives. Methods for conducting evaluation should be practical and clearly understood 
by members of the organization. It is important to consult with organization members in developing the design 


of an evaluation. 


Evaluation provides information which should inform participants of choices about the future of the 


organization and should be seen as contributing to opportunities for change. 


The evaluation process should be: 
(1) Constructive in nature, and useful to organization members in re-directing the 


efforts of the organization or revising its vision, 


(ii) Self-evaluative; that is, members of the organization own the process and its 
results. Evaluation must first serve their need for measures of effectiveness 


and success with the goals of the organization established by members. 


(ii) Participatory by design and include in the process useful opportunities for 
input from all elements of the organization and those affected by its work. 
This includes Board and committee members, regular members, 


consumers/clients, staff and funders. 


(iv) Routine and regular; evaluation must be conducted on a regularly scheduled 


basis rather than when convenient or in response to crisis. 


2. Describing the Context for SPRC Evaluation 


Enhancing organizational planning and priority-setting within the SPRC requires a recognition that all 
activities are evaluated. Given the complex nature of SPRC activities, it is also important to recognize that there 
is not one template or model which is most appropriate for evaluation. Each activity may require a 


fundamentally different mode of assessment. 


=] 


In addition, there is a need to consider the definition of how much time and effort can be prescribed 
to a particular SPRC evaluation activity, A "rule of thumb" is that 3-5 percent of activity time should be taken 
up in evaluation. However, the actual amount of time should more likely relate to the amount of corresponding 


time spent in carrying out the overall activity. 


Every evaluative component is, at the very least, dependent on: 
a) the parameters of the program or activity; 
b) SPRC goals and values against which it should be measured, e.g., “effective 
citizen participation", or "providing support to community decision-making"; 
and 


c) utilization of SPRC Principles as a basis for beginning any such process of evaluation. 


3. The Evaluation Process 
Any process should seek to ensure community involvement and to make certain that community 


stakeholders have a clear understanding of evaluation within the context of that activity. 


In formulating effective evaluation activities, it is important to be inclusive in determining participants 
and explicit in the identification of roles, vis-a-vis which human resources should be utilized in the evaluation, 


i.e., staff, Board/Volunteers, participants, etc. 


It is also important that the organization track its processes in a comprehensive manner, tying this 
concept back to our ongoing priorities and planning system. Using the philosophical models which have been 
established, 1.e., community development tenets, Statement of Research Philosophy, and evaluation principles, 
this tracking should enable the organization to look both inwardly at effectiveness and to utilize information to 


validate our external accountability to both the "community" and funders. 


It is critical that the SPRC be clear in stating its objectives behind each involvement in the community. 
It is important to acknowledge that the decision-making process within which the organization functions really 


tends to be more political at times than practical. 


action: 


FUTURE DIRECTIONS 


The future direction of this report and its findings should consider the following recommendations for 


a The organization (SPRC) should formally adopt the report of the Evaluation Task Force, 


including the framework document which outlines "Principles for Evaluation". 


2, Develop some policy statements which serve to incorporate and integrate the findings of the 
Task Force, including evaluation principles, into the structure process and outcomes of SPRC 
activities, 1.e., any new proposal should have an evaluation component noted within its related 


documents. 


3; Further identify and articulate the vehicles or means by which these evaluation-based policies 


can be most effectively implemented and measured. 


4, a) At the outset, (e.g., in the first six months) these policies should be 
implemented using just one category of SPRC activities, e.g., In-house 


° 


projects, and a means of "pilot-testing." 


b) Staff should report back to the Board in approximately six months re: 


development and implementation of these policies. 
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APPENDIX A 


EVALUATION TASK FORCE MEMBERSHIP LIST 


Mary Ann Seldon (Chairperson) SPRC Board 

George McLauchlin SPRC Board - Community Development 
Marion Emo SPRC Board - Research Advisory Committee 
Doug Sider Research Advisory Committee 

Linda Jenkins Community 

Don Jaffray SPRC staff 

Ritch Whyman Community 

Staff Resource: Caroline Ball 


Mike Pennock 


APPENDIX B 


INVENTORY OF PROJECTS AND CONSULTATIONS 


Fall: 1991 


Social Planning and Research Council of Hamilton and District 
155 James Street South, Suite 601, Hamilton, Ontario. L8P 3A4 
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I. FUNDED PROJECTS AND CONSULTATIONS 


1) Settlement and Integration Study 


PURPOSE: To conduct a needs assessment of settlement and integration services for 
immigrants in Hamilton-Wentworth. 


Staff Contacts: Mike Pennock and Denise Brooks 
Completion Date: February, 1991 
Comments: 1.5 Staff persons have been hired to carry out this project (Denise Brooks 


and Akosia Tulloch). Board approval of final report will be required. Steering Committee has 
been formed. 


2) Hamilton Against Poverty 


PURPOSE: To provide organizational development assistance, technical and resource 
support to Hamilton Against Poverty. 


Staff Contact: Don Jaffray 
Completion Date: August, 1992 


Comments: Three staff-persons have been hired through the Provincial Social Services 
Employment Project. 


3) Housing Assistance and Awareness Program 


PURPOSE: To increase the awareness of the affordable housing problem among the 
labour community. This project is in its third year of operation. 


Staff Contact: Don Jaffray and Paula Randazzo 
Comments: Although this project will finish in January, it has resulted in the entry of 


the United Steelworkers’ Local 1005 into the non-profit housing field. Ongoing support will 
be provided under a new project entitled "Labourhood". 


4) Citizen Action Group Evaluation 


PURPOSE: To evaluate a new plant-closure program offered by Citizen Action Group. 


Statft- Contact: Mike Pennock 
Completion Date: March 31, 1992 


Comments; SPRC will be working in partnership with a private sector consulting service 
in completing this evaluation. 


5) Regional Employment Assistance Program Evaluation 


PURPOSE: To evaluate a new employment assistance program in Regional Social 
Services. 


Staff Contact: Caroline Ball 
Completion Date: March: 31. 1992 


Comments: Consultation initiated by Employment and Immigration Canada and 
Regional Social Services 


Il. NON-FUNDED PROJECTS AND CONSULTATIONS 


1) Promise the Children 2000 


PURPOSE: To develop a comprehensive community plan for alleviating the prevalence 
and consequences of child poverty in Hamilton-Wentworth. 


Staff Contact: Mike Pennock 


Completion Date: Plan development: February, 1992 
Implementation: Ongoing 


Comments: Initiated by SPRC. Network of external committees developed. 


2) Taxation Review Task Force 


PURPOSE: To develop SPRC policies pertaining to taxation and related issues. 
Staff Contact: Mike Pennock 
Completion Date: February, 1992 


Comments: Internal SPRC Committee 


3) Community Agencies Review Board 


PURPOSE: To develop a mechanism for reviewing consumer complaints about 
community services. 


Staff Contact: Don Jaffray 
Completion Date: December, 1991 


Status: Consultation initiated by SPRC and Regional Social Services. 


4)'Sel€ Help Centre 
PURPOSE: To develop a Centre for Self Help Groups in Hamilton-Wentworth. 


Stati Gontact: Mike Pennock 
Completion Date: August, 1992 


Status: External committec initiated by SPRC. Implementation of recommendation from 
1988 study of self-help groups by SPRC. 


5) Community Services Planning Grou 


PURPOSE: To develop a co-ordinated approach to planning and priority-setting among 
community services in Hamilton-Wentworth. 


Staff Contact: Mike Pennock 
Completion Date: Ongoing 


Status: External committee initiated by SPRC. 


III. MEMBERSHIP ON EXTERNAL COMMITTEES 


1) Addictions Committee of the District Health Council 


PURPOSE: To facilitate the development of a service co-ordination model for addiction 
services in Hamilton-Wentworth. 


Staff Contact: Don Jaffray, committee member 
Chairperson , Addiction Services Planning Committee 


2) AATD Advisory Planning Team 


PURPOSE: To provide advice to staff on the use of four major data bases in the 
planning of services and identification of trends in Hamilton-Wentworth. 


Star Contact: Gloria DeSantis, committee member 


3) United Way 
a) Agency Relations Committee 


PURPOSE: To plan for and allocate United Way funds to agencies in Hamilton- 
Wentworth. 


StatteGontact Caroline Ball, committee consultant. 


b) Allocations Working Group 


Staff Contact: Caroline Ball, committee member 


4) Central Area Plan Implementation Committee (City of Hamilton) 


PURPOSE: To advise the City Planning and Development Committee on downtown 
development issues. 


Staff Contact: Gloria DeSantis, committee member 


5) Educating Future Physicians in Ontario: Public Expectations Committee 


PURPOSE: To define the future role of physicians in Ontario and make appropriate 
revisions in medical school curriculum. 


Staff Contact: Mike Pennock, committee chairperson 


6) District Health Council - Mental Health Committee 


PURPOSE: To provide a forum for communication and information exchange among 
service providers and consumers in order to facilitate the planning and co-ordination of a 
comprehensive mental health system in Hamilton-Wentworth. 


Staff Contact: Don Jaffray/Gloria DeSantis, committee member 


7) Regional Chairman’s Sustainable Development Task Force 


PURPOSE: To apply the concepts of sustainable development to Regional Government 
decision-making. 


Staff Contacts: Caroline Ball/Gloria DeSantis, staff consultants. 


8) Labour Force Constellation 


PURPOSE: To develop a plan for meeting future labour force needs in Hamilton- 
Wentworth. 


Staff Contact: Mike Pennock, committee member 


9) Regional Employment Adjustment Committee 


PURPOSE: To develop an effective response to plant closures and major lay-offs in 
Hamilton-Wentworth. 


Staff Contact: Mike Pennock, committee member 


10) Long Term Care Consultation Advisor mmitte 

PURPOSE: To advise the Provincial Community Health and Support Services Division 
on the development of an effective consultation process for long term care services in 
Hamilton-Wentworth. 


Starr Contact. Mike Pennock, committee member. 


11) Anonymous Testing and AIDS Committee 


PURPOSE: To advise the Region about the development of anonymous testing for HIV+ 
and AIDS in Hamilton-Wentworth. 


Staff (Contact: Mike Pennock, committee chairperson 


12) Hamilton-Wentworth Housing Authority: Skills Development Program Steering Committee 


PURPOSE: To develop and implement specialized skill development programs for 
children in non-profit housing. 


Staff Contact: Caroline Ball, committee member 


13) Community Information Services Planning Committee 
PURPOSE: To assist CIS in service planning. 


Staff Contact: Caroline Ball, committee member 


IV. PROJECTS AND MAJOR CONSULTATIONS COMPLETED SINCE FALL 1990 


Hamilton Public Library Locke-Picton Study 


This paid consultation was completed; HPL is now in the process of implementing results. 


Hamilton Foundation Priorities Study 


This paid consultation was completed. Report available. 


Long-Term Care for the Disabled 


SPRC was represented on a committee of the Regional Municipality which reviewed the 
needs of disabled persons in Hamilton-Wentworth. The Final report has been completed and 
will go to the Region in the Fall of 1991. 


Diverse Racial and Cultural Groups Access to the Social Services System 


Carried out under our funding agreement with Region. Final report has been approved by 
the Board of Directors. SPRC is monitoring implementation and has established internal 
committee for the development of SPRC policies. 


Regional Chairman’s Task Force on Affordable Housing 


° 


Task Force has completed its work. SPRC provided staff support and assisted with an 
evaluation of Task Force process. Letter has been sent to the Region suggesting that Task 
Force continue to have a role in monitoring implementation. 


Regional Indian Centre - Employment Needs Assessment 


Funded consultation. Final report prepared by the Regional Indian Centre. SPRC provided 
research services. 


Regional Task Force on AIDS 
SPRC staff served on the Task Force. Report has been presented to the Region. Currently 


in implementation stage. SPRC staff-person is chairing a committee dealing with the 
implementation of anonymous testing for HIV+ and AIDS in Hamilton-Wentworth. 


Sexual Abuse Task Force 


SPRC staff served on external Task Force. Final report has been prepared. No immediate 
implications for SPRC. 


r Y h Implementation mmitte 


SPRC staff served on external committee which was set up to implement results of Street 
Youth Task Force. Implementation has been completed with the establishment of a new 
program for street youth. 


Victims’ Services Task Force 


SPRC provided staff and logistical support to an external task force which was created to 
establish a support service to victims of crime. Funding has been secured and an agency is 
being established. SPRC is serving as a trustee of funds until the service is properly 
constituted. SPRC staff person (Mike Pennock) is serving on the Board of Directors of this 
agency. 


Kirkendall-Strathcona Neighbourhood House Needs Assessment 


SPRC provided staff support to Wesley Urban Ministries in carrying out this needs 
assessment. The report has been completed and Wesley Urban Ministries is now in the process 
of implementing the recommendations. 


City of Hamilton Housing Intensification Study 


SPRC provided staff support to this study. The report has been completed and adopted by 
City Council. 


October 15, 1991 
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APPENDIX C 


STAFF TIME ANALYSIS 


In general, there has been an increase in administrative time with a corresponding decrease 
in project and consultation time. Background or professional development time has also 


decreased while time commitments to community liaison have increased. 


Administrative Activities 


Within the time-recording system, administrative activities are divided into two categories - 
general administrationand administrative committees. The latter category includes attendance 


and preparation time for all of the administrative committees of the Council. 


Both of these categories have increased. The substantial increase in the allocation of time 


to administrative committees has resulted from a number of factors: 
1) An increase in the number of staff meetings. 


2) The establishment of new administrative committees during the past 
two years - Priorities Review Task Force, Sustainable Development 
Sub-Committee, our Multicultural Sub-Committee and the Transportation 


Sub-Committee. 


3) An unfilled vacancy for part of the last fiscal year, with no 


corresponding decrease in administrative commitments. 


The reasons for the increase in general administration are somewhat less clear. Activities 
relating to our ongoing strategic planning activities fall into this category, as does the time 
devoted to preparing unsolicited funding proposals (i.e., the Evaluation Unit). Both of these 
activities have increased during the past two or three years and have undoubtedly had an 
impact. However, we do not record the nature of specific administrative activities within our 


time recording system, so a more definitive analysis is not possible. 


86/87 87/88 88/89 89/90 90/91 
Administration 
a) General 10.0 13.6 17.4 n/a 16:5 
b) Committee 8.5 8.6 8.8 n/a 12.0 
Sub-total 18.5 222 26.2 n/a 28.5 
Program Committees 55 3.9. 4.7 n/a 25 
Consultations 
a) Major 23.5 [33 20.9 n/a 2.) 
b) Minor 16.1 ee) 16.8 n/a 1955 
Sub-total 39.6 25.2 oie A n/a 45.0 
Projects 27.9 Syed 22,4 n/a 16.5 
Background 6.8 a3 4.8 n/a 4.0 
Community Liaison 1.6 1.8 4.2 n/a 3.5 
Total 99.9 100.1 100.0 100.0 


Program Committees 


Program committees are standing or ad hoc committees which are created for the purpose 
of guiding program activities (i.e., Research Advisory Committee, Community Development 


Advisory Committee and Social Housing Action Committee). 


The recent decrease in the time allocation to this category is largely due to the inactivity 


of the Community Development Advisory Committee. 


Background 


Professional development activities have decreased, largely as a result of diminished time 
availability. These activities tend to be "squeezed out" by increases in other categories. There 


are also very limited resources available for conference activities. 


Community Liaison 


The majority of activities in this area involve preseatations, speeches and liaison with the 


media. Time allocations in this category have increased. 


Projects and Consultations 


Projects and consultations comprise the "program" or “service” area of SPRC activities. 
Activities are categorized as consultations if they are carried out in response to external 
requests and are not "owned" by the SPRC (i.e., the SPRC does not own the final report or 


product of the activity). Projects are internally generated activities. 


For administrative purposes, consultations are divided into two categories: Major 
consultations involve more than 35 hours of staff time in a 12-month period and require Board 


approval. Minor consultations involve less than 35 hours of staff time and do not require 


Board approval. 


The allocation of staff time to these categories has decreased in response to increased 
administrative allocations. Within this category, there has been a significant reduction in 
project time and a slight increase in consultation time, particularly in major consultations. 


Annual fluctuations in these categories cause difficulty in identifying specific trends. 


Projects and major consultations involved approximately 42% of staff time in 1990-91. For 
the purposes of this analysis, the individual projects and major consultations were classified, 


by the Executive Director, in terms of: 


a) the primary content area addressed (i.e., housing) 


b) the primary functional area of activity involved. 


The "functional areas" were defined on the basis of the definitions which had been utilized 


in a 1982 priorities study within the SPRC. 


The attached figures present the results of this analysis. Figure I indicates the percentage 
of total project/major consultation time that was devoted to each of 11 content areas. Figure 


II presents the same breakdown, classified by functional areas. 


It is important to note that these figures only apply to SPRC "core staff" time - project staff 
time is not included. In the case of "illiteracy", for example, the Council was involved in a 
major consultation on this topic but relatively little core staff time was devoted to it. We had 


a full-time project staff person who carried out the work. 


FIGURE! 


Projects and Major Consultations by Content Area 


% 
1) Housing 2527 
2) Poverty/Income peered 
3) Multiculturalism 167 
4) Child’care 6.6 
5) Taxation 3.4 
6) Addictions eh! 
7) Employment 25 
8) Disabled WOE 
9) Mental Health lied 
10) Illiteracy 13 
11) Cross-sector 13.0 


FIGURE di 


Projects and Major Consultations by Functional Area 


% 
1) Social policy development and advocacy 52:2 
2) Service co-ordination 13:9 
3) Organizational assistance to community groups 1097 
4) Social impact assessment of government policy 10.3 
5) Socio-economic trends analysis 5.6 
6) Needs assessment 43 
7) Evaluation 25 


8) Public policy changes affecting citizen participation .6 


APPENDIX D 


Sample Matrix of SPRC Activities 
by Methods of Evaluation 


Key Components Means of Evaluation Notes 
A. Policy 
Generation 
1. SHAC - action/advisory - process analysis 
- development of - impact assessment 
strategies re - participant feedbk. 
attitudes and 
community educ. 
- committee support 
& participation in 
dev. of housing 
awareness activities 
2. Taxation - time limited - content analysis 
Review - review of taxation - process anaysis 
Committee issues: - delphi process 


a) for SPRC policy - participant feedbk. 
b) for public educ. 
c) for discussion 


paper/publication 
B. CD-Based 
Planning 
1. Child Pov. - dev. of community - process analysis 
Forum "network" of grps. - storytelling 
focusing on impact - impact assessment 
of future strategic - network analysis 
plans in/between 
sectors to decrease 
child poverty 
2. Multi- - community-based - process analysis 
cultural consultation to - feasibility assessment 
Study look at access of - participant satisfaction 


Givesracial/ cult. surveys 
groups to soc. ser. 


Cc. Research 


1. Publication - dev. of written - content analysis 
Development materials for use - community consults 
by the public - cost benefit analysis 


Trends 


Networks/ 


- collection and 
analysis of data 
focusing on future 
trends, e.g.,labour 
and economic dev. 


Community Liaison 


Sustainable - SPRC in-house comn. 


Dev. 


URIs 


- staff involvement 
in work grps. and 
Process Advisory 
Committee 


systems analysis 
content analysis 
community circulation 


participant satis. 
staff/volunteer retreat 


T.F. self-evaluation 
process analysis 
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